Clustering Videos by Location

Location is a useful source of information for a variety of tasks. Just as users may want to tag and search their personal photo collections and videos for specific people, they may also want to specify a location to further narrow down the search. Users may also want to browse videos by location, annotate locations, or create location specific compilations. We propose an algorithm that uses visual information to cluster video shots by the location in which they were captured. We demonstrate our algorithm on both home videos and professionally edited footage such as sitcoms [1, 8]. In the context of home movies, location generally means a specific room in the house, or a frequently visited place outside, such as in the garden, or at the local park. In the context of sitcoms, location means a film “set” such as the coffee shop in the sitcom “Friends.” Our algorithm first breaks the video into shots using a simple color histogram-based algorithm [3]. It is important to fully represent the visual varieties in each shot. We empirically compare three approaches: (1) Using a single keyframe, the middleframe of the shot. (2) Using multiple keyframes sampled uniformly in time from the video [7]. And (3) Stitching the frames into a mosaic [1]. We illustrate these three choices in Figure 1. We found the second approach to perform the best. Next we need to measure the similarity between each pair of keyframes in the shot representation. We considered two approaches: (1) bag of visual words based [2]. (2) feature matching based [6]. We found the first approach to perform far better than the second. The next step in the design of our algorithm is the core clustering algorithm. Again, we considered several choices: (1) k-means, (2) a “connected components” algorithm, (3) a spectral clustering algorithm [5], and (4) a model-based algorithm [4] using an energy function that is specifically designed to model the expected shape of clusters for the task at hand. We found the final approach (described in Section 2) to peform the best. As subsequent shots in a video are likely to have been captured at the same location it is reasonable to incorporate this prior knowledge into the clustering process. The final component in our algorithm is to add a temporal prior, which significantly improves performance, particularly for professionally edited video. We provide quantitative empirical evaluations on both home videos and professionally edited content (4 episodes of the sitcom “Friends”) to justify each choice made in the design of our algorithm. These evaluations are performed using manually-specified ground-truth location labels.